Anyone in a sinful condition or nature is a sinner before God. The whole world is under the law and held accountable to God (Romans 3:19, 23). No one can be saved without Jesus (John 14:6; Acts 4:12). Even if infants die before they are born, they still need Jesus to go to heaven.
All of us including infants are sinners. When infants die, I believe they become innocent, because the wages of sin is death (Romans 6:23). However, that doesn’t mean they become righteous. Only the ones who have the righteousness and holiness of Jesus Christ can be in heaven. “It is because of him that you are in Christ Jesus, who has become for us wisdom from God—that is, our righteousness, holiness and redemption” (1 Corinthians 1:30).
God chose His people in Christ before the foundation of the world (Ephesians 1:4). Jeremiah 1:5 tells us that some infants are set apart before they are born, which implies some are not. Also, remember example of Jacob and Isaac: “Though they were not yet born and had done nothing either good or bad—in order that God’s purpose of election might continue, not because of works but because of him who calls” (Romans 9:11).