Despite his successes, Asa's life was marked by personal sorrows. The loss of his children at early ages quite possibly led him to many of his philanthropic ideals. Considered nouveau riche by his contemporaries, Asa never truly forgot his humble beginnings, and in many respects, left the world far better than he found it. Regarded as quiet and reserved, his generous deeds spoke for him. A philanthropist throughout his lifetime, Asa gave $33 million dollars to the town of Mauch Chunk and to the Lehigh Valley. A year and half after celebrating his 50th wedding anniversary, Asa passed away. A fall at his office in Philadelphia had left him bedridden for several weeks in which time pneumonia had set in. He would enter into rest on May 17, 1879. He left behind his beloved wife, Sarah, and three children - Mary, Robert, and Harry. At the time of his passing, Asa retained an estate valued at $54 and a half million dollars.
Mr. Packer's associates issued the following statement at a Lehigh Valley Railroad Company Board of Directors meeting after learning of his passing:
The man of iron nerve, whose life was one of purity and uprightness, simple in habit, dignified in demeanor, fervent, earnest, free of all forms of ostentation, liberal beyond measure, to whose magnanimity of soul hundreds of living witnesses pay heartfelt tribute, had passed away May 17, 1879. The Directors of the Lehigh Valley Railroad Company have heard with profound sorrow of the death of their President, the Hon. Asa Packer, by which each one of the Directors has lost a true and valued friend, the company has lost its founder and its sagacious leader, the laboring man has lost a sympathizing benefactor, and our country has lost a useful and patriotic citizen. ~ As published in The Lehigh Valley Railroad Company's Annual Report, June 10, 1879