— June 2, 2017
Project Evaluation —

Project Evaluation

 

Kelcie Feeney, Yumi King, Rosanna Lui, Natalia Rodrigues

 

Problem Statement: Bees are misunderstood creatures that are feared by many people due to their ability to sting. The bad experience and misconceptions about bees make it hard for people to care about bees and be more empathetic towards them. Empathy towards bees is important for their and our survival.

Goals:

  • To put users out into nature where bees are, so they can learn to watch, observe, and understand bees
  • Users will create a connection between the plants and bees
  • To be immerse Beetrice’s world where the user would have fun playing the game.

Audiences:

  • The main audience 7 to 9 year olds but with adults that might be interested

Technologies:

  • Augmented Reality

 

Study Plan and Methods Used

For our study plan and methods used, we used the following:


Questionnaires: Developed two questionnaires for user testing; one for “Bee Sweet to the Earth” and another for “Beefest” at West Seattle Bee Garden” (all questions found below)

Research Questionnaires: Created two surveys for beekeepers and people with little to no experience with bee to find more information between the two groups

User Testing: Conversed with non stakeholders during Bee Fest and Bee Sweets events for user testing to get information about  (individual beekeepers, troop leaders, professors, and gardeners) along with our target audience, children between the ages 5-7.
Observational Testing: Interacted with children in the target audience and observed how they interacted with the app
Interviews: Met with stakeholders and interviewed them for opinion and ideas on current app progression, audio was also taken from said interviews
Footage: All users whom agreed to disclose their opinions through signing disclosure forms were filmed for future references and information

 

Surveys and Questionnaires Used for the Events
(all replies were short answers)

Bee Sweet to the Earth      BeeFest at West Seattle Bee Garden
How old are you?

Do you have a phone or tablet?

What sort of phone or tablet do you have?

What kind of games do you like to play?

Which face describes how you feel about bees? Why?
Put the face that the child pick for the emoji chart

What did you think of the game?

What did you learn about bees?

Did you find the game fun? Why?

Do you think the game was too easy, just right, or too hard for you to play? Why?

What do you think of Beetrice?

Would you play this game again?

What would you like to do in this game?
How old are you?

What did you think of the game?

What did you think of the character Beetrice?

Did you find the game fun? Why?

Would you play this game again?



Bee Sweet Answer Analysis


Our first user testing session was on April 22nd at 21 Acres. We presented our first prototype of the game in Woodinville at the “Bee Sweet to the Earth” event which featured music, venders, and bee themed events.

This was the first version of Bee Adventurous but it was very incomplete due to time constraints. We decided to try to show the game play of collecting pollen, Beetrice story,  and building a hive. We had a lot of technical difficulties where the  app was crashing when we showed the prototype to people.. We also did not have the educational pages of the app completed, map markers, and the parks button. . Regardless of the lack of features on the app, we tried to get as much feedback as possible to help us improve the app. While at Bee Sweets, we tested on 27 users of various ages. 11 users were adults the 16 were children.

One of our main goals of the game is to change the negative stigma associated with bees. In order to test this we had to first quantify the user’s feelings about bees. We did this by having the user point to a sheet that list emojis feelings from excellent to not good. Since we were working with children, we decided to add faces to the scale to help the child recognize what emotion they felt about bees.  We felt that the data would be skewed since we were presenting at an event that focuses on promoting local food systems and education about the environment including education on bees. Any children at this event may have already been educated by their parents or guardians about the importance of bees. If we were to test at another event that was not focusing on the importance of bees and environmental action we may have gotten more negative emotions associated with bees.

At the event, 33% of the users tested said that they feel between good and excellent about bees. 29.6% of the users tested felt excellent about bees and 29.6% felt less than good about bees.

 

Notable Observation:

 

  • The age demographic was all over the place where we had kids that were way too young or old.
  • Many people were very interested in Beetrice as a character and thought she was a male or needed a male companion
  • Majority of the testers skipped the beginning story to get to the gameplay
  • The younger the kids the more interested they were in naming the hives/queen
  • Younger children preferred the smaller phones instead of the larger tablet
  • Majority of the children there struggled with reading and had their parents do most of the engagement and survey questions
  • The survey questions were too long and we lost their attention after 3 to 4 questions in.
  • People were interested in Beetrice in the story
  • The tapping intrigued people and got them slightly more immersed into the game
  • Most of the kids didn’t have a phone and was often given the phone by the parents
  • Majority of the kids were more interested in the merchandise and didn’t realize it was a game
  • We lost people’s interest due to load time of the game
  • The highest is 66 years old and the lowest is the 3 year old.
  • 6 people over 30
  • 8 people dont play game.
  • Majority of parents said their kids were too young to play games
  • 44% had an iphone.

 

Quantitative Data

 

While showing the users the app we recorded what observations we were noticing. Some children were very young and needed guidance from an adult, one child was extremely nervous and another ran away. We were also noticing that the users were becoming frustrated with the app crashing. We decided that it was very important to figure out why the app was crashing because it was making the game unplayable.

 

After the user was done playing the game we asked them what they thought about it. We received a lot of positive feedback. One negative feedback we got was that the game may be too hard for children under our target age range. We also got feedback, this game might be too easy for some children who are older than 9 and it needed to be more challenging.

 

We wanted to make sure that we were creating a game that was playable for children so we also asked the users if the game was too easy, just right, or too hard. 6 users said that the game was just right. 8 users said that the game was easy. 2 users said that the game was just fine and one user said that the game was too hard.

 

One way that we were trying to build empathy for bees is by telling the story of Beetrice. Beetrice is the main character of Bee Adventurous. In order to build a connection with the audience we had to have a main character that the user found relatable. We tested this by asking the children what they thought about Beetrice after playing the game. We got all positive replies. most users thought that she was cute. Which was great because we are really trying to push the cute factor. One user was unsure about how they felt about Beetrice. A handful of users also assumed Beetrice was a male or felt that the game needed a male character.

 

We really wanted to make sure that we were heading in the right direction with this game. So we asked the users if they would play the game again. We were expecting a lot of negative replies since the app was crashing and basically unplayable but surprisingly we got back all positive answers, one maybe and one no. We asked why the user would not play the game again and their response was that they “do not play games.”.

 

Stakeholder Interview

 

After Bee Sweets, we had interviewed two more of our stakeholders to show them the more complete version of the app. Both of them approved of the game and had a lot of positive feedback in which they think it will be educational. Both of them felt the game accomplishes the goal of empathy but the game needed to be built out more.

 

Pacific Science Center Sarah Moore:

Does the game play make sense?

 

Yes

Do you think this would be educational?

Yes

How can this game be more educational?

It is a bit of a mix. I think people will just want to play and start learning afterward.

Do you think this game would change children’s perception of bees?

Yeah, any time you imagine being something you motivated to learn more about it and be less scared

How do you feel about the visual style of the game?
I like the flower, the bee looks too young so make it older unless we go younger.


What do you think of the character Beetrice?

I like her

How do you think kids will respond to Beatrice?

I think they will expect her to help them someway.

What would you like to do in this game?

It should have a dance section possibly Beetrice showing the dance. Earned fun facts when you collect pollen when you gained it

Are there any specifics that you want to add?

None

 

West Seattle Bee Garden – Lauren:

 

Does the game play make sense?

understand the concept, not enough time and content yet.

 

Do you think this would be educational?

yes

 

How can this game be more educational?

work on the facts that come up, what they are doing is happening in the real world, the impactful fact

 

Do you think this game would change children’s perception of bees?

yes

 

How do you feel about the visual style of the game?

love the look

 

What do you think of the character Beetrice?

like the name, is cute and fun

 

How do you think kids will respond to Beatrice?

think they will like her, like the goal, seems friendly, fact about stings and dies to show had to sting to protect and scared it,

 

What would you like to do in this game?

on the right path, share it with a group of beekeeper, educators

 

Are there any specifics that you want to add?

involve human beekeeper helping with that fear, coming to check on us to see we are okay,

 

 

BeeFest Answer Analysis

 

After BeeSweets, we built the game further and presented a more stable version with a working gps map to the public at BeeFest held at West Seattle Bee Garden.

While there were more children present at the event, they tend to leave before we were able to ask questions; but one thing was evident and that was they greatly enjoyed the app to the point where they brought their friends for the demo. Compared to the random crashes of the previous version, this rendition of the app crash once or twice and the adults were very understanding whilst the children simply moved on to the next phone or asked us to fix it to play it some more.  Although we still didn’t have all the planned features implemented, the children seemed to have fun with the current product and the adults naturally suggested what we were already planning (plant identifier, bee identifier, ect). In short, Bee Adventurous was successful in terms of enjoyment and excitement for the finished product.

The issues faced here was the same as before. The children that were learning about bees were the children that liked bees in the first place and already felt empathetic towards them.

 

Notable Observations:

  • The parents or guardians with multiple children were more willing to sign the documents in comparison to those with singular amount
  • People were very interested in Beetrice as a character
  • When the game broke, children were more patient to see it fixed whilst the older audience gave the device back
  • Target audience was more willing to figure the game out by themselves whilst older audiences waited for directions
  • There was competitive play if there were more children around (who could get the higher level)
  • Majority of the testers skipped the beginning story to get to the gameplay
  • Younger children preferred the smaller phones instead of the larger tablet
  • 60% of attendents were children 10 and under, and 42% were target demographic
  • Older attendents expressed interest in showing game to younger acquaintances
  • The younger the kids the more interested they were in naming the hives/queen
  • Half the people were amused by Beetrice’s name; most of whom were adults
  • Most children liked bees prior to playing the game
  • Most people suggest game features that we were already planning to implement

 

Quantitative Data

 

Positive Response: 91% (22/24)

Neutral Response: 9% (2/24)

Negative Response: 0% (0/24)

Positive Response: 100% (24/24)

Neutral Response: 0% (0/24)

Negative Response: 0% (0/24)

 

Positive Response: 70% (14/20)

Neutral Response: 15% (3/20)

Negative Response: 15% (3/20)

 

Questionnaire Analysis

In total, we documented 52 people testing our game. 27 of this number came from “Bee Sweet to the Earth” event and the remaining 25 user testing came from “Beefest at West Seattle Bee Garden.” Our target demographic was children around the ages of 5-7; although there were more children testing our game at “Beefest” they tended to run away as soon as the experience was done, thus failing to record their answers. In comparison to the smaller population at “Bee Sweets” we were able to record more as the people were more relaxed and reluctant to stay indoors to answer (it was raining outside). One consistent element in the testing was positive feedback on our game as they thought it was cute (especially Beetrice) and educational. Although the adults were less likely to find it fun.


Due to the fact that both events were at festivals, a lot of questions were reduced to simple answers such as “yes” or “no” (especially from children) as they were interested to spend time at other booths as well. At “Bee Sweets” it was rather evident that the users were getting bored of the plentiful questions and so we reduced the number of questions for “Beefest.”

Heuristic Evaluation
Team’s Review: Team Smoggy

During the time of the evaluation, the app was unstable and clearly incomplete due to coding issues so a lot of concerns about the app was either addressed and fixed or already in development. That being said, some of the biggest concern found on the evaluation includes readability and intuitiveness of the layout, technicality issues, and lack of game mechanics. Below is the overall review for the game.

Overall Review: Overall, quite impressed! The game works which is impressive considering the technical restraints and ramp up skill set I’m sure you experienced. With that being said, I would suggest to focus more on text/font and overall design structure. While the
technical features are working (bugs all all! But to be expected,) the experience would benefit from some focus on overall design


Feedback Examples:

1.1.4 Are high informative contents placed in high hierarchy areas? Does Not Meet
I would say the text displayed could be more strategic in where it’s placed, also focus on font and size. Very hard to read!

1.4.5 Are low discoverable areas as touch buttons well identifiable? Does Not Meet

 

2.1.1 Are metaphors properly used as visual clues? Meets

It meets, but there are some bugs. For example, when it “broke” there was no indication of an error screen, it just froze, no directions on how to fix

 

2.1.4 Do the selected colours correspond to common expectations about colour codes? Does Not Meet

I would say the text displayed could be more strategic in where it’s placed, also focus on font and size. Very hard to read

3.1.1 Can users move forward and backward between fields or dialog box? Meets

It meets, but there are some bugs. For example, when it “broke” there was no indication of an error screen, it just froze, no directions on how to fix

 

3.1.2 If the system has multi page data entry screens, can users move backward and forward among all the pages in the set? Meets

No pop-up window displaying error message – would be really helpful though!

 

4.2.3 Is the number of colour used constrained up to four? Are additional colours saved for occasional use only? Meets

Consistent Color Pallet

 

4.5.3 Are user actions named consistently across all prompts in the system? Meets

Consistent Usage of Words


 

Summary of Findings

Analysis

Overall, the public seemed to enjoy the concept of Bee Adventurous and many of them like to see it progress to the app store. Children enjoy the ability to interact and click on flowers when they are outside whilst the adults enjoy the education aspect of the app. Many of the adults, especially those that have children or take care of them were happy to hear that we were planning to place in a plant identifier. As one adult stated, “it’s hard to find a game where the child isn’t aware that they are learning, if lessons are pushed onto them, they get bored.”
From observation, a lot of people tended to skip the intro story in order to get to the gameplay and the ones that were interested was our target audience… but many of them either lacked or were learning the ability to read. But that inherently is not a bad thing as the parents started to lean down and read it out loud to the kids. In other words, other than the potential of teaching children about plants and bees, there is the potential to teach children how to read.

In terms of the devices, many children chose to interact with the phones whilst the adults chose the larger ipad. The reason was because children could hold the phone more easily and the adults chose the larger screen to help their children and guide them through the app.
All of them thought that the bee, Beetrice, was really cute. At least three people stated that they emphasized with her and three children cared about her wellbeing.   

Lessons Learned

At “Bee Sweets” we had a long questionnaire but with short answers and at “Beefest” we had a short questionnaire with open ended questions in terms of length. During the first festival, there was a hesitancy to answer questions and an impatience to get the questions done after the third question, especially from the adults. This impatience was greatly reduced in “Beefest” when we told the amount of questions and asked the children while they were being entertained by the game. Most people were willing to give critical feedback, but not many would stay to do so.

Children are either shy or very self expressive, hence the need to change attitude according to the individual child. Most kids replied with a “yes” or “no” question, despised the question being opened ended. Having the parents help their child answer the questions proved to be more fruitful in terms getting critical feedback.

One other observation was the fact the overall consensus about bees were positive. Our project was to spread empathy for bees and whilst our project will to do well in terms of marketing, we need a wider range of audience that didn’t come in with a positive viewpoint on bees. Should we choose another event to attend to in the future, we would choose an insect event that’s not focused on bees in particular. That way we can potentially get a group of children that are interested in bugs, have a fear of bees, and willing more to learn more about it.

 

Future Plans

Our future plans includes old ideas that were placed in the back burner and new ideas that came from observing the children interacting with Bee Adventurous.

 

  • Accompanying audio to story elements: Since our target audience is young, many exclaimed that they couldn’t read the story hence why they skipped straight to the game play. So to help them along, we wanted an audio segment where children and read along and enjoy the story.
  • Seasonal Changes in game: We wanted to implement seasonal changes in the story to promote seasonal challenges that the bees face which includes food shortage and hostile temperatures elements.
  • AR Camera: We want children to take pictures of fauna and maybe include Beetrice as a model in the pictures. This way, the children are more motivated to go outside and look at plants.
  • Competitive/ Cooperative Gameplay: A lot of children showed interest in competing with each other and helping each other. Specifically, they wanted to see who can collect the most pollen and have the highest level and they also want to help each other build hives.
  • Make the Game available for the IPhone: 44% of the children we tested had access to iphones instead of Androids. We feel that we should develop for the IPhone to allow access for these children.

 


Bee Adventurous at Bee Sweets — April 22, 2017
Yumi – Pollen Collection Through Tabbing In Unity — March 13, 2017
Yumi-Camera Device in Unity —
What Each Group Member Did? — March 11, 2017

What Each Group Member Did?

Kelcie

  • Worked on Geolocation
  • Logo
  • Interviews
  • UI design
  • Paper prototype
  • Gaming system (current)
  • Elevator Pitch
  • Project Backlog
  • Interviews
  • Outreach to stakeholders
  • Empathy research
  • tutorials

 

Rosanna

  • A lot of Interviews
  • UI design
  • Interactive Prototype
  • Paper prototype
  • User Flow
  • Working on building the database to store information (current)
  • Making the UI in Unity
  • Color Palette
  • Elevator Pitch
  • Made presentations
  • Storyboard Beetrice story
  • Project Backlog
  • Paper Prototype Video
  • Outreach to stakeholders

 

Yumi

  • Camera working in unity
  • Tapping for pollen collections (current)
  • Transcribed interviews
  • Interviews
  • Did research on bee culture and fun facts
  • Outreach to stakeholders
  • Paper Prototype Video

 

Natalia

  • Bee model design
  • Interviews
  • Storyboard Beetrice story
  • Competitive Analysis on Bee game
  • Outreach to stakeholders
  • Integrated bee model into Unity
  • Three bee animation tied to the tapping
Natalia – Augmented Reality Model — February 22, 2017
Rosanna – User Flow of Game Play —
Screenshot of Broken scoring system —

Screenshot of Broken scoring system

screen-shot-2017-02-22-at-2-06-10-am

This week I began working on the scoring system. I am feeling pretty lost because I do not have the other components of the game to actually work with the interactive score. Right now I am looking into creating the game without the assets so that I can test that it works correctly, then we can add images in later. – Kelcie

Yumi – Camera In Unity —