"The two companies have now jointly come to the conclusion that these talks will no longer be continued," Porsche said in a statement on Friday.
Porsche said the two companies could not agree on a partnership that included both an engine partnership and also the F1 team.
The automaker said taking part in F1 remained an attractive option and it will continue to monitor the possibility as F1's rule changes are finalized.
Volkswagen Group's former CEO Herbert Diess said in May that Porsche and Audi would join Formula One, which has so far been dominated by German rival Mercedes-Benz.
Media reports had suggested long-standing talks between Porsche and Red Bull were stalling due to a gap between how much control the VW Group-owned marque wanted and what Red Bull was prepared to give.
Red Bull principal Christian Horner ruled out a Porsche takeover last Friday and said any partnership would have to be on the Formula One team's terms.
Volkswagen-owned Audi said in August it will build an engine in Germany and enter in 2026 with an existing team, likely to be Sauber. F1 has been waiting for Porsche's move.