Homework-3

Lecture7: Ideate

Enhancing human abilities in the age of Augmented Reality.

Idea 1:

Visualizing dream in the VR environment

Can we ever have an ability to visualize the dream again after we wake up? I always dream of watching my dream after I wake up from the sleep. Is this technologically feasible? what are the obstacles to build a system in which a person can replay the dream he saw last night? I am going to present a simple idea to illustrate this novelty.

A quick review on how we see things in our environment

Light enters into eye through the cornea and finally reaches the retina. Retina is a light-sensitive nerve layer where the image is inverted. The optic nerve is then carry the signals to the visual cortex of the brain. Whenever we see objects/images a set of neurons generates spikes in the visual cortex. This phenomena can be recorded via Electroencephalography (EEG) or neural interface microsystem: 100-element silicon-based MEA via a custom hermetic feedthrough design (Ref: 1).

Illustration

To illustrate further, lets start with a simple picture containing a cat. When we see a cat, a set of neurons fired in the brain. It doesnt matter how many times we see the cat, more of less same sets of neurons/area will be activated in the brain. This signal must be transformed into the computer and machine learning/deep learning techniques must be used to train a system that contains many of the neural activity data for a specific image. So whenever an image is kept in front of an eye, after training the neural network (NN), it will show the possible image in the computer screen (cat/dog). The following table is a simplified version that can be used to train the NN for given know objects and neural activity data.

Figure: Brain signals and corresponding objects.

This is process is very complicated than I present here. But our purpose is to model a simplified version in a theoretical sense. Now if we could predict the object based on the brain signal, this object can be placed in a VR environment. Not necessarily the exact cat/dog we saw in the dream, but the 3D reconstruction of a similar cat/dog. Of course we can reproduce the exact cat/dog we saw in the dream. But our first priority is to recognize the object in the first place while someone is dreaming in her/his sleep.

References:

1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3638022/

2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3306444/

3. https://link.springer.com/chapter/10.1007/978-3-642-45062-4_63

Idea 2:

Brain-Computer Interface for long-term memory management

Brain-Computer Interface is an exciting domain and it will potentially shape the human cognitive ability in the future. A recent milestone in BCI project done by the Neuralink startup is a motivating factor in this domain. Elon Musk has demonstrated a synchronization of a living pigs brain with computer.

Figure: Neuralink's self-contained neural implant functions without the aid of external hardware.

The below figure shows the pig streamed the electrical brain activity being registered by the device. "It’s like a Fitbit in your skull with tiny wires," Musk said in his presentation (Ref: 2).

Figure: One of the Neuralink pigs at Musk’s online demo (Screengrab: Randi Klett)

So in the future, human would be able to control computers simply using the thought process. Thought signals can be used as a command to give instruction to computers and vice versa.

In future we can decide whether we have to store things in our long-term memory or not. For example if a person had a terrible accident, s/he can decide whether to keep that memory or not. So how can we design this in neural level?

Illustration

Hippocampus and frontal cortex is responsible for analyzing sensory inputs according to the current understanding of neuroscience. For example, the taste of doughnut, or a smell of a flower is stored in the brain as a signal (bits of information). This information can be retried, (even though the current technology doesn't allow us to do it and lack of understanding of the brain activity). But this is feasible, only the time is the factor.

Figure: A schematic sketch of the long-term memory management

The above figure is a simple sketch to reach the dream of long-term memory management to enhance the human ability. This will also enhance children's learning ability, memorizing capability, cognitive stability in general.

References:

1. https://www.zdnet.com/article/elon-musks-brain-computer-startup-is-getting-ready-to-blow-your-mind/

2. https://spectrum.ieee.org/the-human-os/biomedical/devices/elon-musk-neuralink-advance-brains-ai

3. https://www.braingate.org/

4. https://science.howstuffworks.com/life/inside-the-mind/human-brain/human-memory1.htm

5. https://qbi.uq.edu.au/brain-basics/memory/where-are-memories-stored

6. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6132650/

7. https://www.bioserendipity.com/neuron-and-glia-remodeling-contribute-to-male-behavior/rat_human_brain/

8. https://brainome.ucsd.edu/

Idea 3:

Robots/human with 3D printed bio-organs

Can we ever will have ability to reproduce human internal organs? Many people die every year because of the internal organ failure. If we could propose a right technology, then definitely we can enhance the human capacity to live a longer life. Bio printing is an exciting area of research. In which the cellular level structures are built from stem cell technology by enabling bioinks to the structures. Already many laboratory are producing human like bio printed miniature organs such as human livers, hearts etc.

https://www.youtube.com/watch?v=64-ujrKUxLc

Video: How soon are 3D printed organs coming?

3D bio printed eye is another useful idea that can help one day to replace human eye with an alternative bio-printed eye. So what kind of material can be used? The advancement of material science help us to invent new materials, such as Polymeric Material that could be used in tissue engineering and cellular transplantation.

References:

1. https://www.3dnatives.com/en/bioprinting-projects-3d-printed-organs-070420205/#!

2. https://pubs.acs.org/doi/pdf/10.1021/bk-1984-0256.ch001

3. https://www.materialstoday.com/polymers-soft-materials/news/material-repairs-traumatized-organs-and-tissue/

Lecture8: Prototype

According to Scientific American, it is indeed possible to measure brain activities during the time of dreaming. I will go one step further and try to understand the content of the dream and re-visualize in the VR environment. Hence, I will use the Idea 1 to build a prototype, a sketch that can be used to build the whole system in the future. This is a simplified testing, in which we use only two images cat/dog.

Step 1:

Take several brain neural activity data by showing images of cats/dogs to participants.

Step 2:

Carefully record the spikes using EEG or any other nano electrodes.

Step 3:

Map between the images of cats/dogs with the corresponding brain waves. Pixel level mapping will help to reconstruct the exact image.

Step 4:

Train the ML model for the possible prediction of a new (unknown) brain signal.

Step 5:

After obtain the predicted image (cats/dogs) from the brain data, a 3D image reconstruct can be used to obtain a 3D model and game engine like Unity can be used to develop a VR model.

Step 6:

Immersive visualization of the dream in the VR environment.

Below figure describe an architectural design of visualizing an object in the VR environment directly from the neural data.

Figure: A complete schematic sketch to reproduce the image from neural activity data.

As I explained before, we take neural activity data when the person is awake by showing different images. When the person is sleep, we measure brain activity and try to map the dream based on the obtained brain data.

References:

1. https://www.scientificamerican.com/article/brain-activity-during-sleep-can-predict-when-someone-is-dreaming/

2. https://www.leeloosesotericorner.com/dreamsymbol-analysis.html

3. https://webkid.io/blog/neural-networks-in-javascript/

4. https://www.arxiv-vanity.com/papers/1804.06375/

Lecture 9: Use the Prototype for Evaluation

I have discussed about this idea with my friend to get a critical evaluation. He said the idea is good, and can also be used to animate the cat in the unity environment. Because usually in the dream, things will have motions. Moreover, he asked the question of what if the other objects in the scene and possible interaction with the environment? My answer is as follow;

This is a primitive architectural design to implement and visualize the dream. I assume my world consists of only cats and dogs, no other objects. This assumption is to simplify the model solving ability in a constrained environment before extending to a large scale.

There are many obstacles to implement this idea. First of all neuroscience is a still developing field and many mysterious/unanswered questions out there about the function of neurons and brain activity, memory in general. Well this doesn't stop us achieving this goal!! For example, we do not know how exactly water (H2O) molecules behave in a complex fluid flow, we still didn't solve analytically the equation of Navier–Stokes (which is one of the millennium-problems). However, we are good at solving fluid problems numerically using computers, and to forecast the behavior of fluids for the short time interval. Similarly it is not necessary to understand the brain functions fully before we interact with the brain in the neural level. Right technology with right people will help me to achieve this goal one day. :)