The future of music experience: AI, immersive worlds and virtual sound

Currently, a whole new generation of music consumers is growing up who experience music and media very differently, says Thijs Verhulst, Business Development Manager at Warner Music. "If you look at younger audiences, you see that young people aged 16 to 18-years-old spend a lot more time on visual media like YouTube Shorts or TikTok. The group that comes next spends a lot more time in gaming worlds. Both their attention span is shorter, but also the expectations of an artist are very different. They expect something they can see, but they also seek a connection based on more than just music." So technology is not just a possibility, but a necessity in the changing landscape. We already see examples where technology is used successfully, says Verhulst: "Think of the virtual ABBA show that runs successfully, or a concert by Travis Scott in the game Fortnite that attracted 27.7 million visitors - and gave a boost to its chart listing. Closer to home, artist Coloray won prizes for his audiovisual project 'Future Static'. Anything is possible and you can already see the change." 

Until now, immersion has mainly been used in marketing activities. Presenting yourself broadly as an artist is ultimately key to success. These are also logical opportunities, Verhulst explains. "By promoting music with more than audio, you simply increase visibility, but you can also stand out in a market where there is a lot. In addition, it is very important that you can connect with fans on a level of shared values, which creates bonding. A big part is pioneering, especially when it comes to an integration into being an artist. It's a blank canvas in that sense." 

Great artists are already playing around with the possibilities. Drake, for example, has a bizarre shop, where you can look around his house 'The Embassy' and click on products. This way you buy the same fridge magnets as the rapper - from which he probably makes some money. Several stars explored the possibilities of VR, and Dua Lipa and Ellie Golding used AR in promotional campaigns. Verhulst expects that in the coming years this technology will be increasingly embraced by the music world. Moving along and exploring possibilities are therefore very important for artists themselves: "If you as an artist are not working on this, it will be difficult to get your career off the ground in a few years. Now you mainly see big artists who do a lot with this. That is why it is also about looking for what works for you as an artist, and how you can link a revenue model to it. You can't do everything, like a global star like Dua Lipa who has the support of a huge team for this. There is no one-size-fits-all model, every artist has to find what works for them."

Not only the big stars are exploring the boundaries of technology, in Eindhoven there is also a lot of experimentation going on. To make technology accessible to artists, Effenaar founded Effenaar Lab in 2018. Effenaar Lab functions as a makerspace, where artists gain knowledge and can experiment with VR, AR and streaming. Incredibly valuable pioneering work, for the artists and the industry, says Verhulst. "It is very important that we make the knowledge about what is possible available to artists. For me, it's also a kind of benchmarking; find out what artists their stance is and how they think about this. Because they are given the creative space, many original ideas arise, from which I get inspiration."   

One of the trajectories within Effenaar Lab is the Hybrid Music Vibes program. Last March, the second edition of the program started with a new group of artists. After a series of knowledge sessions with experts, the artists got to work themselves and developed their own concept. Of these, six artists were chosen by a professional jury and the public to realize that concept. Together with specialists, they are working on their own project to strengthen their artistry. Many possibilities, but also many questions: How can you strengthen fan experience and connection? How can you enrich audiovisually? And how do you use technology in a cost-effective way? Artists RHIANNIN, De Toegift, Julia Sabaté, Max Frimout, Hiigo and PXPLE JAZZ explore the answers to these questions with their concepts.

Julia Sabaté is not so much a band. It is a project by Julia Sabaté, Ruben Sieben and Koen Hermans, of which Sabaté is the face. The trio is very future-oriented, they see technology as a tool to strengthen their (live) act. Sabaté has Spanish roots and often sings in Spanish. In her opinion, this creates a distance between her and the Dutch concertgoers. "It makes it more difficult to convey the story, although listeners can often feel and experience the emotion. Our idea is to use visuals for that." The group joined forces with Bureau Moeilijke Dingen to further develop this. The first ideas varied a lot, but in the end it was decided to use generative artificial intelligence for the visuals. The audience can submit images, which are fed to the algorithm, making the audience co-creator of the show. This is then applied to one song in the setlist, says Sieben. "You can't have people sent pictures for a whole show, that's going to be way too busy. We are looking for a VJ and for one song it will generate visuals with the submitted images. In retrospect, you could also generate revenue with that image, but we want to look at that in the future. The most important thing is that what we're doing now is something we can do well and strengthen our act." 

The group has a strong pro-technological stance, which they say is part of how Julia Sabaté operates and approaches the artistry. Sabaté sees Hybrid Music Vibes as a reality check regarding the future: "So much is possible. You have to want to understand what's coming and be able to place that in your profession and what opportunities it offers you as an artist." On stage, it's still about playing the songs well and delivering a strong performance. Technology helps with this, and especially enriches the toolkit that you have as an artist, says Sieben: "As an artist, you want to keep up with the times. That's sometimes in your style, but also in terms of technology. For example, artificial intelligence is now super current and offers many opportunities, but you have to know what it can do. Technology can take a lot of work off your hands, like in our live set, but as an artist you're still in control." 

Technology offers possibilities during shows, but also outside of them. It gives several options to stay in touch with fans or to present yourself more broadly and personally as an artists something that appeals to PRXPLE JAZZ, Hiigo, RHIANNIN and De Toegift.   

The Dutch band Hiigo started off with an idea to bring their artwork to life with VR/AR glasses during live shows. After the knowledge sessions, the group concluded that a different course was better, says singer Hans van der Werf. "That was of course unfeasible in terms of costs and organization. Then we were immersed in the world of holograms and metaverse." From the original idea, only the artwork of Laura Gardebroek – the sister of drummer Koen – remained in the new plan. "Our central concept is togetherness, the feeling of home. We want to create an environment where our fans can experience that, watch how we do things and how seriously we take it. For example, how all the artwork by Laura is made with real dedication, that we work with passionate video makers, and how everything is intertwined. I don't like the flat, commercial representation of artists. I just want to show authenticity. That's why we're looking for opportunities in our project for depth and connection with our fans."

Together with partner organizations, a virtual house is being developed. After purchasing a physical product, fans get access. "You can view and explore the artwork, but also listen to demos in Koen's office. There are also holograms of us, which are being developed by Dutch Rose Media, and hopefully we can offer even more in the future. It's going to be very special." The band wants to continue to develop the house into a meeting place for fans and themselves. "A community of fans and musicians who love what they do". 

Singer PXRPLE JAZZ also wants to create a stronger bond with her fans with her concept. The singer sees music as part of the bigger picture and that is the performance. "I see myself as an artist who creates an atmosphere and move. I am very visual in this and that happens naturally to me. This means that I look at the lights, visuals, dance and dancers, but also the message as a whole. All the skills I have come together to make something." It certainly helps to have a broad skillset, and to continue to broaden it as an artist. 

She was already aware of the technical possibilities, and previously experimented with visuals and 3D at live shows. But this time she is going for the realization of an immersive website. "You can try things yourself, and that's very valuable, but you don't want to miss the mark when you hire professionals. It is therefore very important to ask yourself what works for you and what you want to achieve with it. For me, that was connecting with fans. I am active on social media, but everything is fragmented. I'd like to bring that together." Through gamified elements, users can discover more about PXRPLE JAZZ in a playful way. For the realization of the website she works together with arfected and her own team. The website is still in development, so no spoilers, but the goal is clear for Jazz: "I hope this will help fans to feel more connected with me, but also to have easier contact with each other."

With the project RHIANNIN, singer Rianne Wilbers wants to express herself in all facets of her work, which she translates within Hybrid Music Vibes to a virtual diorama. "My vision is an aesthetic overall picture, in which Icelandic artist Björk is my great inspiration. Her work is instantly recognizable in everything she does, from her sound to clothing and artwork. RHIANNIN is that to me; An identity in which my art, music and myself form one image." In collaboration with arfected, Wilbers is working on an immersive environment, where people can experience her music and paintings and walk through them. Wilbers is not a techie herself, so she made a book with cut-out pages as inspiration to illustrate what she had in mind for the technicians.   

Transforming your art into a world with a virtual character also means asking questions about your work. "A lot of image formatting and generation can be done with artificial intelligence (AI), but I want to keep the feeling of authentic craftsmanship and make sure that my style comes back. There's a lot of jokes in there, and surrealism, so I want to come up with that myself and not generate it. But for example, I had a painting of a bird, where one wing is cut off and you can complete that for the website with generative AI. So much is uncharted territory for everyone, but it is a huge enrichment of my artistry and the journey of discovery for me of what RHIANNIN is and will become."

The Dutch band De Toegift also creates a virtual world, but also takes into account the character of their work and being. "We're actually an old-fashioned band, so exploring the possibilities technology offers within Hybrid Music Vibes program is special. Our performances have little fuss; We are a group of musicians on stage and we play vulnerable music that draws on folk, jazz, pop and rock. This creates a vulnerable and intimate relationship with the audience, which we want to preserve. But we stepped in with an open mind," says keyboardist and guitarist Tom Gudde. De Toegift is enthusiastic about how many aspects and angles technology offers, but it has to match who they are. "I don't see De Toegift using LED screens or VR/AR applications that disrupt the experience between us and the audience." 

However, there is still a lot to be gained for the band outside the stage, which they hope to find an answer to with technology. "Our music touches people and we see that they often stick around to talk to us, but that time is limited." The plan became to develop a virtual environment, which fans can access via a QR code at shows. Here they can ask questions, share emotions or experiences and thus build a connection with the band. Gudde hopes that this environment will help to strengthen the relationship with fans, and that is precisely in the qualitative one-on-one interactions with fans. "With this project we create a safe environment, where we can take all the time for that contact a day or week later. I am always very curious about what people have to say and I also like to receive new stories. That goes for the rest of the band as well, and it fits who we are as well. We hope to use this as a pilot during our autumn tour and we are very curious to see what it brings us."

For the realization of the interactive environment, De Toegift works together with augmented reality experts Dutch Rose Media. Sparring about the choices in the project got Gudde thinking: "We had the first kick-off and it is very interesting, because you have technicians who have all the possibilities and trends on top of their mind. My role as a creative means that I have to give it a purpose and style and that means making choices. Sometimes also against the trends. One example is that they indicated that everything is video nowadays, but I opted for a text-based interaction. There's a romance to letters and the written word that fits De Toegift." Gudde is aware that De Toegift does not opt for an application that uses the most 'cutting edge' possibilities. He is certainly aware of the changes that are coming, but above all sees the need for customization: "We are a different kind of band." 

Most people think of virtual reality as visual, but artist Max Frimout thinks this is a limited view of what VR means. He finds sound much more interesting and how it interacts with your physical positioning as a listener: "There are all kinds of applications that play with a virtual environment. This can be a treadmill that gives you the feeling of movement, but also tactile interaction. If you are going to watch sound from a virtual space, then anything is possible. I find the spatial aspect of sound very special." 

During Hybrid Music Vibes, Frimout will further develop the concept of the Odio app. Odio was developed by a team of innovators, including Frimout, for the AirPods, according to Frimout one of the best VR devices available. The app allows the user to create a kind of live remix of how sound is coming at them. But how that really works is complicated. The musician picks up an old mixing console: "Look, reproducing sound in stereo used to be completely new. Before that, you had mono, where the sound comes from one point. But with stereo sound, new ways of setting up a sound experience arose, because you have two points. I have an old stereo mixer here, with which I can adjust how all the elements in that stereo sound come to me as a listener and in what balance. That's also the idea with virtual sound in the Odio app, where the speakers make it possible to simulate sound from all sides of the room around you. As a user, you can place all sound elements in the space around you as you wish." 

To further develop the concept of Odio, Frimout works together with iOS developer Unbeatable and Odio media. In that respect, it is pioneering work, in which a lot is possible, but where the sound artist also challenges his own artistry: "For myself, it is fascinating to investigate where sound goes, but also to broaden and deepen myself. It is very special to compose for Odio, because we normally think - in what I call linear structures. But also think of video games, where sound has to work fragmented based on a user's randomness. That's also the case with virtual sound. It would be great if I could eventually do live performances with Odio in that way." 

The app is already available and became a great success with 150 thousand downloads and even won an Apple Design Award. Why is Odio so successful? Frimout sees several explanations: "Playing with sound intrigues people, something with being in control. There is also a dimension of personalization to it; Decorate the sound the way you like it best. Anyway, now the app is still a 2d interface, but within the project I want to see what else is possible, now that Apple glasses is coming. For me, it's also a question of whether virtual is just a new tool, or really a game changer in how we think about making music."

Curious about the Hybrid Music Vibes program? Here you can find more about the program and the participating artists. Follow the Instagram of Effenaar Lab if you would like to stay up to date of cool projects and see the results of the experiments.