Shelley Duncan hit a three-run homer and Ben Zobrist had a pair of RBI singles for the Rays, who twice came from behind before wasting a late two-run lead.
Fernando Rodney (1-0) blew a save opportunity by allowing a run for the first time since last Aug. 18. Brian Roberts' RBI double off Rodney in the ninth made it 7-all.
Chris Davis went 4 for 4 for the Orioles, hitting a three-run homer for the second straight day and driving in four runs. He also had an RBI single and a double.
Joyce, who had a bunt single as a pinch hitter in the seventh, won it by homering on a 2-1 pitch.
Duncan's first hit for his new team finished a four-run sixth that wiped out an early 4-0 deficit. The Orioles regained a one-run lead in the seventh, only to watch the Rays answer with three more runs to go ahead 7-5.
But Baltimore wasn't finished, scoring on Matt Wieters' RBI single in the eighth and Roberts' tying double.
Pedro Strop, the second pitcher to follow Orioles starter Wei-Yin Chen, was unable to hold the 5-4 lead Baltimore took on Nate McLouth's run-scoring single in the seventh.
Joyce bunted for a single in the seventh and later scored on a wild pitch. Pinch hitter James Loney's put the Rays ahead and Zobrist's second RBI single of the night made it 7-5.
Davis' homer came off Rays starter Jeremy Hellickson, landing in the mostly empty stands in left-center field. After drawing a sellout crowd of 34,078 on Tuesday, attendance was announced as 15,599 for the middle date of a three-game series.
The 27-year-old Chen won a team-leading 12 games as a rookie last season, the most wins by an Orioles pitcher since Erik Bedard won 13 in 2007.
Desmond Jennings, Evan Longoria and Jose Lobaton led off the first three innings against the left-hander with singles, but each of them was erased by double plays.
Down 4-0, Tampa Bay broke through in the sixth. Zobrist chased Chen with a two-out RBI single and Duncan's homer made it 4-all.