Pilot Study (Dec 2020)

Overview#

In December, 2020 and January, 2021, we performed a simple pilot study to obtain user feedback on the InternAloha user interface and functionality.

Two InternAloha developers each interviewed two ICS student subjects over Discord, for a total of four interviews. The developers led the subjects through the site, asked them questions, and filled out a form to gather information during the interview. Unfortunately, two different versions of the form were used during the interviews, but useful information was still obtained.

All subjects completed an Informed Consent document prior to the interview. The Informed Consent document is available here.

Results#

Here is a summary of the results, which have been lightly edited for clarity and consistency.

Question 1: Before I show you the InternAloha system, I would like to learn about your experiences so far with internships. First, do you have any prior experience looking for internships? If so, when did you look and what happened?#

I have prior experience looking for internships, I've looked around the university career fair, LinkedIn job board, levels.fyi and cold mails. Of the internships I got offered/accepted, I found them through the career fair.
I haven't really looked at any internship sites so far since it's not my top priority.
Looked at USAJobs, and Intelligence Careers for internships. I have applied for a few.
No I haven't. Highest priority is getting through the difficult ICS classes currently.

Q2: Are you planning to look for an internship in 2021? Why or why not?#

I want to get some non-IT related experience that I feel could make me a much stronger candidate when I start applying for jobs. (I am security track)
Not yet as I'm doing a double major right now and searching for one seems to be a bit time consuming currently.
Not really as I want to when quarantine fully finishes to start searching them out.
Not looking for an internship in 2021 since I will be graduating.

Q3: If you are planning to look for an internship in 2021, how do you plan to look? What sites might you use online? Are there any resources at UH that you might use?#

Most likely through the different known places like Linkedin, Indeed, etc. I'm not so sure of any UH resources to use as I haven't really seen much for myself but maybe SECE if they do internships.
Whether it is paid or not. What skill will I be able to learn from those internships. Know what I'm getting into in terms of responsibilities and possible opportunities (scholarships, potential job openings, etc). Perhaps Radgrad would be one, but I'm not fond of the amount of opportunities it currently has.
Not planning to look, but if I did, I would use the UH career fair, Gerald's email list, ACM discord chat, linkedin and levels.fyi. Found them to be effective in terms of getting an internship.
I plan to keep looking at USAJobs and Intelligence careers. The only other resource is Student Opportunity Center. I have set up a profile but have not used very much.

Q4: What are the top three things you would look for in an internship listing to see if it was right for you?#

If it pays or not, it doesn't really matter what kind of company it is until you actually work there. I would also prefer to know what is expected from me before and when working there, as well as maybe location (remote or not).
Interests is at the top. Would like to know if the internships are paid or not. Locations is also a priority to know. Interview options and knowing what will be on the interview would be great.
Location (on or off island, remote etc.), paid (I'm not considering un-paid), and with which department. I'm mainly looking at ones with the federal government.
match to personal interest, location and length

Q4: Now let's look at InternAloha. Please go to: https://internaloha.github.io/internaloha/. Please share your screen with me so I can see what you are doing with the system. Please try to search for internships of interest. Are you able to easily check for the top three things of interest?#

For the paid/un-paid I would have thought that this information would be included either in the title header, or I could find it via a filter. The results do seem relevant and UI for the most part is intuitive.
Yes I can see pretty much everything that I wanted to see here, but some of the job listings don't have a specified expectations portions, which is fine.
I don't really see the paid or not clearly but everything else more or less shows.
UI is pretty intuitive, the listings seem to be pretty good.

Q5: How would you recommend we improve the InternAloha system to make it more useful for you?#

Each user made many suggestions. I have made each suggestion a separate bullet point for readability.

The summary description is very disorganized at a glance.
The layouts feel everywhere and have no set format. It's better to see a couple of internships after the filter and get out.
The layout is nice right now, but there should be a lots of fixes in having a succinct description pattern and perhaps having less listings per column (2-3).
The posted and last updated tags don't need to be up there, perhaps in the detail description or even at the bottom.
The most important information should be put on the top and then the specifics can be searched for after (In a sense, too much information shown in the top description).
Being able have and hold an account to the listings you liked (even cookies could be a choice).
Perhaps having a questionnaire that ties into the search algorithm before showing the listing because the direct listing is very daunting to look at in first glance.
Having a main page would be fantastic to have as well to sum of what the website is for since it's a little confusing to understand what the website is without being told.
Having a pop up for hovering on favorite and report button.
Having a submit button for the search functions.
Having a way to reset the database when jobs are search because there isn't a real way to go back to the default listing (like a clear button).
Have a hover popup for the sort by options to explain them (particularly date and the difference between internship and company).
Paginated pages to be able to see options from newest to oldest, etc.
Having a search count for the career interests like in skills.
Career Interest: distrusted systems.
A little cluttered, too much scrolling.
Maybe when you click on the card, it show something on the side.
Would be nice to have a US only option
Filter based off of government/non-government. They know this is a unique need but others may find it useful if implemented.

Q6: Thank you very much for participating in this study! Is there anything else you'd like to add?#

Seems weird to display the bubbles of skills that a job doesn't have.
I would rather simply include a list of the skills required/wanted per position. Rather than all of them and then just grey some out. (I later explained that it's meant for when users create a profile and list their skills. Blue are the ones they have and gray are the ones they're missing but I can see why it's weird/confusing when profile hasn't been implemented yet)
It has easy-to-navigate filtering by having exact interests and qualifications to choose. Really nice design and color choice. Most people don't have time to go through all the descriptions so if there is a way to shorten them. I don't see a point in looking at them if the websites have them anyways.
Overall, the website looks really nice and the filtering system works great, but I'd only recommend this website as a low priority option at its current state.
Site looks pretty good, don't really have anything else to add.

Q7: (FOR INTERVIEWERS ONLY) Please record any additional thoughts about this interview below. Are there ways we could improve the system and/or pilot study protocol?#

We can do a lot of reorganization for the job listing items such as the posted and the scrapped because it seems like people like to see what is the internship about more. For Zip Recruiter specifically, some of the hyperlinks bring a person to the homepage instead of the direct listing. Some hyperlinks also might have changed to new listings as well, so we need to check whether each scraper does go to the right listings when clicked on. Inconsistencies with the CSS of the descriptions. Numerous duplicate job listings (Python Developer (Intern)). Expired jobs. Remove Bavaria, and State from locations.
There seems to be discourse of whether to have the scroll down or popup descriptions, but one thing for sure is to find a way to shorten the information to the most important stuff and also maintaining a succinct CSS for the descriptions.
Bug with infinite scroll. If users zoom out then try to scroll down, it doesn't always load new listings.

Discussion#

My major take-away is that we are on the right track, but we need to do a better job of filtering. Here are a couple of ideas:

Redesign the home page as a questionnaire.

We need to know whether various filters are "must have", or "good to have". For example, some students only want to see paid internships. Some students only want US jobs. I believe we can improve the UX by having students start by creating a kind of profile where we find out about "required" vs. "desired" attributes of the internship. Once they've filled this out, they can submit it to see results. That way, we don't inundate them with so much information right away.

The home page can also provide some details on what sites are scraped, how many listings we have, etc.

Compute a "score" for each internship relative to the user profile, then present "acceptable" listings in descending order by score .

The system might be more impactful if it helped the student to understand the level of match with their profile.

First, if a student indicates an attribute (such as "local", or "paid") as required, then only those internships that satisfy those required conditions are "acceptable" and presented in the results. From the acceptable listings, a score is computing based upon the level of match. A score of "100" would be a perfect match.