A research and development project with the aim to develop a platform that provides architects, video game creators and designers with innovative tools to enhance and simplify the creative phase of the designing process. The main idea behind V4Design is to reuse (i) visual: movies, documentaries paintings and images from other artwork and (ii) textual content: from textual documentations in films, critics, catalogues, museum guides, and re-purpose it in order in a way that will be useful for architecture and video game designers.
V4Design developed a data collection and retrieval tool that can gather data from content providers and crawl on-line art libraries, in order to extract 3D and VR representations from objects, buildings and cityscape environments. Additionally, V4Design introduces innovative design tools to architects, designers and video game creators that will leverage visual and textual ICT technologies: (i) extract a specific historic era’s artistic or aesthetic style, localize buildings and art-objects in visual data, (ii) generate personalized summaries of the retrieved commentaries, reviews, critics, etc. on the visual content, and (iii) enhance all the above with semantic knowledge, smart indexing and retrieval capability.
INFRASTRUCTURE INNOVATIONS
V4Design focused on a few key innovations to support the reuse of the 3D cultural heritage content.
V4Design Crawler: The project developed a tool that collects digital cultural heritage content from relevant web sources and social media platforms in a single module. It takes different inputs and outputs them as a single interoperable metadata standard: SIMMO. It applies various techniques (including scraping, retrieval from API) to integrate the most popular websites and platforms that share openly licensed cultural heritage.
Aesthetics Extraction: Project partners used machine learning techniques to extract aesthetic concepts out of architecture and paintings, and train classifiers to recognise certain aesthetic styles in images.
Style transfer: V4Design developed a technique that allows people to recreate the content of an image by adopting the extracted style of an image depicting a painting, advancing this technology over the timeframe of the project.
Spatio-temporal building and object localisation: Using Computer vision techniques, V4Design developed machine learning models that can detect and annotate several parts of images and videos. The models can easily distinguish the facade of a building from its background and surroundings, and then tag the different parts of this building with textual metadata tags. A separate scene recognition module, characterises the images or video frames as indoor or outdoor scenes. If the scene is outdoors, then the surroundings of the building are automatically analysed for more information. If the scene is indoors, another model tries to locate and annotate objects that might be indoors: furniture, decoration, etc. These models all work together to assist the 3D reconstruction of these identified objects by removing unwanted clutter and increasing dense reconstruction performance. For this purpose, deep learning models were trained and extended, using specialised datasets fully aligned with the V4Design’s scope.
3D Model Reconstruction: The project developed an automated video processing pipeline for pre-existing video footage which makes consecutive frames suitable for reconstruction and separates video from multiple angles. The project also used an alternative method for handling medium to large sized image collection datasets, involving a pre-trained feature vocabulary to detect similar images in a much faster manner. By using the inputs of the style transfer module, the project could restyle the texture of the models.
V4Design authoring tools: Through the project, people using Unity or Rhino3D, two widely used and popular 3D editing applications, now have access to a plugin that makes re-using cultural heritage as easy as pressing a button. Using either plugin, one can search through the thousands of available 3D models and import them into their project with a click.
Explore all of the innovations in the V4Design project on the project website, where you can find a rundown of all the innovation that took place throughout the project in a useful factsheet.
V4Design was a Horizon2020 project funded as a Research and Innovation Action.
Website: v4design.eu
Project duration: January 2018 – March 2021
PROJECT PARTNERS
The Centre for Research and Technology-Hellas (CERTH) – Information Technologies Institute (ITI) – Project Coordinator
KU Leuven
Universitat Pompeu Fabra
Robert McNeel & Associates
Herzog & de Meuron
The Aristotle University of Thessaloniki
Solaris Filmproduktion / SLRS Multimedia AB
ArtFilms Ltd
Deutsche Welle
Europeana Foundation
Nurogames
