In 1917, Randolph gained his first experience as a labor organizer. He organized a union of elevator operators in New York City. 

In 1919, he was elected president of the National Brotherhood of Workers of America. He was in charge of organizing African American shipyard and dock workers in the Tidewater region of Virginia. 

In 1925, he became the president of the Brotherhood of Sleeping Car Porters (BSCP), the first labor organization led by an African American that received a charter in the American Federation of Labor (AFL). Randolph was in charge of organizing the Pullman Company employees. The Pullman Company was a manufacturing company of railroad cars founded by George Pullman. The railroads had expanded during this time, leading to plenty of work. With the widespread discrimination, the Pullman Company was major in the Black community since they hired many African Americans. But because they were not unionized, George Pullman took advantage, offering poor working conditions, little pay, and long hours. 

Randolph took on a huge task that would require many sacrifices. But his role as president led him to great success. However, it did not come without turmoil and many trials. With the help of labor and civil rights activists Rosina Corrothers Tucker and other porters, Randolph was able to enroll 51% of the porters within a year. But once Pullman caught wind of the porter’s enrollment, they were fired and met with violence. As a result, many were afraid and dropped out of the union. 

The union did not see real change until 1932 when President Roosevelt was elected. Once the 1934 Railway Labor Act was passed, the porters were granted rights under federal law. The Brotherhood membership skyrocketed to over 7,000 participants. In 1935, the organization began to negotiate with the Pullman Company, and in 1937, they came to an agreement. After years of struggling, employees were now able to enjoy a $2,000,000 pay increase, overtime pay, fewer hours, and shorter weeks.