Pilot Study (May 2021)

Overview#

In May, 2021, we performed a second pilot of InternAloha after a semester of development. This pilot study had a similar structure to the December 2020 Pilot Study, and it's worth reviewing those results to see what has changed (and what has not changed) in the past four months.

Four InternAloha developers each interviewed two ICS student subjects over Discord, for a total of eight interviews. The developers led the subjects through the site, asked them questions, and filled out a form to gather information during the interview.

All subjects completed an Informed Consent document prior to the interview. The Informed Consent document is available here.

Results#

Here is a summary of the results, which have been lightly edited for clarity and consistency.

Question 1: Before I show you the InternAloha system, I would like to learn about your experiences so far with internships. First, do you have any prior experience looking for internships? If so, when did you look and what happened?#

  • Yes, it was really hard to find an internship for Physics major. The hardest part is to find one that fit my need. Started in Sophomore year, it was not accepted. Because I did not know much upper level knowledge and did not have much experience. Then I did an internship with my school in Junior year.
  • Yes, one of the experiences was looking for an ICS job from SECE, but otherwise hasn't looked for internships besides being offered internships. Was offered a full time internship from the job they working in.
  • He went to all the career fairs as well as googling for internships. Internships through emails sent out by the department. He applied to all he could find.
  • He looked for internships online that fits his interests, talk to peers and asked about their experience looking for an internship. Mainly looked during the spring semester before the summer. He didn't look much into internships and wasn't able to get one but he was able to get a research position.
  • Started during sophomore year, applied a few but not successful. Then applied later for school ones and got it.
  • No prior experience as of yet.
  • Looked a little, last time was when they went to the Gray Hopper's Conference about 2 years ago. Said they didn't have a great resume but did get 1 interview and went through 2 of the stages before the pandemic happened and they were all cancelled. They didn't really look at other sites
  • Yes, has a lot of experience. 1st semester in computer science started to look for internships, got rejected non stop. Very first thing was looked up software engineering companies in HI then sent cold emails Linkedin to look for UHM students and their internships and asked how they got their internship and how the process was like.
  • Looked at Linkedin for HR contact to send cold emails. Mainly got in through connections and networking
  • Got internship at Ikyazo through cold emailing (found HR email via linkedin)

Q2: Are you planning to look for an internship in the next year? Why or why not?#

  • Yes, I am. Because I am graduating with Master degree. I hope to get one soon. I want to have more experience.
  • No since they already have an internship at said job.
  • He is not as he is graduating next year.
  • He is looking for experiences such as jobs and not specifically just internships.
  • Yes, plan to apply during summer for spring or fall internships to practice my CS skills and prepare for future jobs.
  • Not really because they want to focus on school first and foremost.
  • Possibly, they are graduating next semester and will also look for a job, whichever one they can get first.
  • No because they're graduating!

Q3: If you are planning to look for an internship in the next year, how do you plan to look? What sites might you use online? Are there any resources at UH that you might use?#

  • I plan to use Chegg internship, Indeed. I am trying find remote internships. One Professor at UH provides internship opportunity. He announces opportunity by email.
  • No because they are graduating next year, so they'll look for a real job instead. For job searching, they prefer using Google Jobs and perhaps Indeed. Perhaps using Radgrad as a UH resource.
  • Although he doesn't plan on looking for internships, he heard about glassdoor and linkedin. UH would mainly be just career fairs.
  • His main way would be asking his peers and seniors. But he might google internships online. He also looks through the opportunities he gets from UH emails.
  • Will use SECE from UH, LinkedIn and job fair if there is any
  • The first plan for an internship is to talk to people in the ICS major if there are possible openings and connections. Next option would most likely be Radgrad and SECE.
  • linkedin, ripplematch, certain companies and look at actual websites
  • If they were looking, go for high reaching companies like FANNG, capable of getting past resume screening. Big tech next
  • Look at their website, and apply every single one

Q4: What are the top three things you would look for in an internship listing to see if it was right for you?#

  • I would look for skills like Python, data science, and any internships that does engineering or scientific disciplines.
  • The pay range, location (remote or not included), and the detailed description of the contents of the job.
  • Paid or nonpaid internship, what the internship entails, like application due dates or what the internship is like, location.
  • He stated: 1. Location 2. Flexible hours 3. Length and compensation
  • The location, the type: skill level, such as programming language, and how much does it pay.
  • Location, schedule flexibility, and of course pay grade.
  • Which field (eg. machine learning vs. data science), location, length
  • Career goals, location, pay

Q4: Now let's look at InternAloha. Please go to: https://internaloha.github.io/internaloha/. Please share your screen with me so I can see what you are doing with the system. Please try to search for internships of interest. Are you able to easily check for the top three things of interest?#

  • Yes, there are a lot of the internships fit my needs.
  • Everything except for pay range, but they understand most websites don't have a pay range.
  • It includes the location, but doesn't include paid or nonpaid. The listings includes the information he wants to know about the internship.
  • He couldn't find hours or length of internship besides in the description. It doesn't specify but he said he understands that normally, it is paid and the description will state not paid if it wasn't.
  • Yes, I believe so.
  • Location is the only thing that is shown for most of the listings, but it makes sense for some websites to not have schedule or pay grade.
  • Can look through pretty easily, like how it is set up. It's pretty easy to use and it pulls from multiple website so it's cool. The listings look pretty relevant, though some of the links clicked were broken and some were outdated (eg. Angellist)
  • Yeah, the listings seem relevant.

Q5: How would you recommend we improve the InternAloha system to make it more useful for you?#

  • Need to check for duplicate posts, some look exactly same. Should add filter for experience, and education level (for undergraduate, graduate).
  • Font colors of the information is kinda hard to see. Boldening them with darker colors will a lot. Since some of the listings are full time jobs, being able to distinguish from internship and job would be nice.
  • He would like a search bar to search up specific things he is looking for an internship. He appreciates the "Last Updated" and the info in the listing descriptions. At first glance he thought the internship listings on top were internships the companies offered itself.
  • He stated there should be an option to specify searching, IE: a listing must have everything he selects. IE: javascript itself has 300 listings and is hard to narrow down when you have multiple things you need. Sort by filter could have a search bar pop up, and then you put in the company name. Have an option to put how many hours the user wants to work per day/week.
  • Maybe change the location filter so people can type instead of choosing/scrolling. Add filter so users can search for certain company.
  • Keep the listings per row to more like 3 maximum. Too many listings at one time.
  • Darken the text font for the starting information since it is too lightly colored. Blends in with the background too much.
  • Perhaps more splashes of color to improve the aesthetic.
  • Implement the ability to add favorites (right now it's just prototype version). Because it is infinite scroll, they would like to have a button to scroll back up, or for it to use pages instead.
  • The UI could use improvement, doesn't look modern. The color, background, text. The cards are okay, but a little too busy. Would be better if you click on the card and a modal pops up.
  • A cool functionality would be to upload resume and parse it to get skills, and have it recommend internships that way.

Q6: Thank you very much for participating in this study! Is there anything else you'd like to add?#

  • The filter are cool, it can arrange by company. Get rid of the internships that are outdated. May extend it for more majors.
  • Being able to have favorites to actually show up at the top will help gratefully. Limit the amount of listings that can show up at a time. It takes a toll on memory. Besides that, the website looks great.
  • They stated that the listings seemed a bit overwhelming, too much words. He suggested that the listing could expand to a bigger card when clicked on. He stated that the website seems a bit dull looking, maybe more contrast? When removing listings it doesn't go back to 0 it results in 2679.
  • He asked a question whether adding multiple tags only included both or included listings that had at least one of them. There were duplicate listings (glassdoor). He originally thought the internship numbers on the top box were internships for that specific company. Looks of the website: Doesn't like how the listings pop up when mouse is hovering over. Would prefer scrolling load, it lags when it shows all the listings. Description boxes are different sizes and some listings have empty space b/c of it. Consistency, text boxes in career interest is capitalized but skills is not. First and second box are different roundness. Feels there is no need to include websites that have 0 internships in the first block. He would enjoy a dark mode.
  • It looks nice and useful.
  • In the skills tab, make the skills capitalized.
  • I don't think favorites button is necessary since people will just pick and choose whatever catches their eye.
  • Containers causes spacing differences because of text sizes (like in location).
  • Extremely useful, would definitely use it. They were actually already looking for internships during the study. Functionality very well.

Q7: (FOR INTERVIEWERS ONLY) Please record any additional thoughts about this interview below. Are there ways we could improve the system and/or pilot study protocol?#

  • I think it is good to hear about how people think about our website. For the pilot study, we should give the volunteer specific number of skills and career interests.
  • Favorite button doesn't work as intended. Doesn't change color.
  • Some random location pins like (N, Not Available, Unknown, and NW).
  • Random array list showing in console (coming from recommendationscript.js).
  • There are some html for images within descriptions, which can cause lag.
  • Occasional crashes of the website for prolonged usage.
  • Error, the paginated paging is broken. New listings are being loaded without even scrolling all the way down.
  • I agree a search bar adds some ease of use. I think there could also be a sort by button like sort by most recent.
  • Perhaps the skills and career interest could be a drop down menu with a checkbox to specify. Another consistency thing is why does career interest not show amount in the tags but skills and location do? The pilot study questions were good, maybe a question can be added like: "What aspect of the website could be more optimized or clear?"
  • I think the suggestion about the filter for company is good and doable.
  • You have to hover over the actual text in career interest to see suggested skill list (hovering over white space doesn't show popup).
  • Duplicates in Glassdoor and SimplyHired (There's around 30 "Summer Intern ? Oncology Bioinformatics")
  • Remove NSF-REU data from the listing pool.
  • Home Page link isn't working as intended (href is there, but button is unclickable)
  • Possible bug in the search feature (location). For example, when narrowing it down from Career Interest > Skills, the number on the states don't match up. Eg. Machine Learning, C++, Arizona (5), but when you click on it, only 1 listing shows up.

Discussion#

For future, it's probably good to avoid non-ICS majors for the pilot study. The fact that a Physics major can't find relevant internships is intentional in the design of InternAloha!

Similarly, for the next pilot study, we should try to get some sophomores and juniors (seniors are no longer interested in searching for internships.) That said, the experiences of seniors are definitely useful to us as well.

We need to improve the quality of the database:

  • Eliminate duplicates
  • Eliminate bogus locations, skills, etc.

We need to ensure that we are providing listings from LinkedIn and GlassDoor.

Along with these things, we should start the process of integrating into RadGrad!