Faith

With Hashem’s help,
 
Greetings and blessings,

One who is not capable of being wicked also cannot be righteous.

So the simplest answer to your question is: Hashem did not create human beings as robots, but as souls with free will. The very existence of free choice is good, because it gives the opportunity to attain the greatest eternal good that can be attained in reality.

Free choice has consequences and results, both positive and negative. Hashem wants that a person choose the good, so that it will be good for the person. But a person does not always choose what is good for him.

"Bread of kindness" (bread of shame) is a concept that describes receiving reward in the World to Come for free, and not through effort, similar to a poor person asking for charity. The idea behind it is that by its nature, the soul is embarrassed to receive status and reward for something it did not invest in, as opposed to the eternal happiness it experiences when it has successfully stood up to tests. In other words, the soul needs to build itself, and to attain its greatness through the tests and the mitzvot, through which it merits closeness to Hashem. The goal is to resemble Hashem, and in order to resemble Hashem, the soul must choose on its own to draw near to Hashem.  
 
A father who loves his children wants his children to grow up and become independent, honest, and righteous people in their own right.
So too, by way of distinction, Hashem wants the souls He created to merit reaching the quality of greatness through their own strength, and thus to merit closeness to Him by right and not by undeserved kindness.
The "game of life" is meant to produce heroes. Heroes not "as if," but heroes in truth.

Regarding your question about free choice -

We have free will, and that is a fact; after all, we ourselves choose whether to keep the mitzvot or to rebel, Heaven forfend, and it is a fact that there are people who return in repentance and people who leave repentance, which you do not find among animals, which have no free choice at all. So even if we do not understand the process of choice and knowledge, this is not a difficulty, since we barely understand even the brain, all the more so the Creator of the world.
And in answer to your question:
Hashem is not a human being, and when it is said that He knows the future, that does not mean human knowledge. Hashem sees the future for the simple reason that He, may He be blessed, is in the future; He is present in every time - past, present, and future simultaneously, since He is infinite and eternal. In other words, at this very moment you are making choices in the past, in the present, and in the future, and Hashem sees all your choices simultaneously - only after you choose them. At every moment you may change your choices - and Hashem will always be aware of what you choose, because He is present in all times. Since Hashem is not bound by time, He does not determine what will be, but merely observes all our choices without interfering with the choice. Therefore, knowledge does not determine the choice; on the contrary, the choice is what determines the knowledge.   

I will be happy to help with any additional question. 

Blessings,
Daniel Bels