It's basic economic theory: scarcity enhances value. If you have an infinite existence, then each individual action becomes less important. Take murder for instance - when you kill someone, if there is no afterlife then you have deprived this person of all of their possible life experiences. You have truly taken something away from them.
If we are all inifinite beings, or all part of the source, then if you kill someone, sure, you are depriving them of certain experiences, but you've given them the experience of being killed, and they will go on to have other experiences that they wouldn't have had had you not killed them. Given the relative little importance of any individual action on such a time scale, you haven't really harmed them.
BUT! Before that gets anyone up in arms, consider the fact that I don't think most people in either group really bases their everyday actions on this philosophical position. Non-believers don't tend to treat every action as meaningless simple because there may be no ultimate meaning. And believers don't tend to minimise each action on each simply because these actions will be relatively unimportant to them a billion billion years from now.
Rather: I think both groups tend to highly value their lives here on earth!