The Julia Edwards Baker Scandal

The Real Story of Mary Lincoln's Troubled Niece

Note: This is part of a discarded chapter from a book I’m working on about Mary Lincoln’s final years. Word limits meant I had to cut pieces of it, which I intend to post here. While I have made edits, the style of writing reflects the fact that it was part of a longer manuscript.

Background

Mary Lincoln’s oldest sister, Elizabeth, once ruled the Springfield social scene, having married Ninian W. Edwards, who dominated the political one. But from 1875 until Mary’s death in 1882, the Edwardes appear in the historical record mainly as her lackeys.

She lived with them following her institutionalization in 1875. In late 1876, and a few months after the expiration of the conservatorship held by Robert Lincoln, she departed for Europe.

On hearing she finally planned to return from Pau, France, in 1880, Elizabeth asked no questions and expected no justifications. She just let her know that multiple rooms had been set aside to store her many trunks, and that she could “roam” the second floor to her heart’s content. She had no unrealistic expectations or admonishments; she simply believed it was best to accept Mary as she was.

It is likely that Mary’s vagaries simply did not faze the Edwardses very much. They had learned the meaning of resignation. By the social standards of the day, they had experienced the worst embarrassment possible, and no eccentricity of Mary’s could come close.

Their attractive daughter Julia had married Edward Lewis Baker, Sr., the powerful editor of the Springfield Journal. Mary wrote at one point that she was annoyed by Elizabeth’s boasting of their impressiveness. Soon that would not be an option.

Elizabeth Edwards, her daughter Lizzie, and the Bakers accompanied the Lincoln party to Washington, with the intent of helping her with hospitalities. But Julia’s behavior soon after Lincoln’s inauguration raised eyebrows, in part because she stayed up late talking with men and drove out with one while her husband was still asleep, and she was soon back in Illinois.[1] 

For the rest of the war, Julia seems to have kept a low profile, but her sister Lizzie eloped with Eugene Clover, a man reputed to be of bad character and a heavy drinker, and her appalled parents refused to acknowledge her for some time.¹ Ninian wrote a querulous letter to Lincoln saying his misery could not be exceeded.

It would be.

Clover was killed (and decapitated) in the Battle of Washita River in 1868. She and her three children lived with Elizabeth Edwards at the same time as Mary, and Lewis, in 1876. Why was Lewis there?

The general understanding is that Julia was sexually promiscuous, and that this, eventually leading to a scandal, was papered over by an offer by then-President Grant, who appointed her husband consul to Argentina, where they remained for many years. Historian Michael Burlingame uncovered a letter written by a friend of the Bakers that summarized the matter: “Mrs Baker was a wayward girl and very attractive woman to the great sorrow of her family and friends. There was a scandal connected with her about 1872, and Mr Baker was sent as Consul to the Argentine Republic where they remained until Mr Baker’s death [in 1897] . . . . He was probably not molested in courtesy to [the] memory of Lincoln, as it was better for them not to return.” The “blow to her mother and father, was one they never recovered from.” [2]

Historians have described her as “not mentally responsible enough for monogamy,” and “susceptible to sexual predators.”[i] While she may have suffered from a bipolar-type disorder that led to periods of hypersexual behavior, she appears to have been viewed as a basically self-sufficient adult. Many years later, well into the twentieth century, a Springfielder remembered the gossip, and said “[Julia] must have been a gay person,” someone who liked to have a good time. That was the impression she got growing up. She archly referred to Julia being “up to her old tricks,” referencing her antics with men, but not referring to a compromised mental state. It’s possible Julia was just more “liberated” than her era could handle.

I was able to uncover the real story by extensive newspaper resarch. I feel somewhat bad about dragging this stuff out after all these years, but it seems important to understanding certain aspects of Mary’s life: the press and cultural norms of the era; her relationships with the Edwardses, Julia’s son, Lewis Baker, and her own son Robert.

While Julia’s actual actions or level of mental responsibility cannot be determined from the newpaper coverage, the contemporary accusation was that she was the predator, though many in the press called out the tendency to blame the woman, saying both were at fault.

The Scandal

It was very much a controversy that played out in the press, though mainly confined to the Louisville, Kentucky area at first. Then the telegraph exchange picked it up, and the Cincinnati Commercial Tribune published full details in July 1872.[ii]

Dr. G. H. Robertson had recently departed the Presbyterian Church in Springfield to take up a ministry in Louisville. He and Julia met in Chicago, took a quick trip to his home, and then returned. Their behavior in transit caused the matter to come to notoriety—whether it was the train ride there or back is unclear, but it seems to be on the return to Chicago. Robertson’s wife was away, and it seems had a history of alcohol abuse and gone on a spree. His intoxicated appearance caught the attention of Louisville passengers, who noticed the “closely veiled” woman representing herself as his wife. The scrutiny apparently led to a quick departure, and en route to Chicago, their affectionate behavior raised eyebrows. A missed train connection mean they spent the night on the tracks—and a stationary train meant that “voices and other sounds could be plainly heard” coming from the few occupied sleeping berths. Shortly afterwards, a reporter approached him for comment about the witness testimony, and the shocked minister did not handle it very smoothly. He claimed to have been taking laudanum and a little brandy for medical reasons, which had rendered him unconscious upon his arrival in Chicago, and he remembered nothing for some time after that  “She had repeatedly requested him to meet her in Chicago”, and when he declined, she said she would come to Louisville and, as the reporter put it,  “possibly rend his clerical garments.”[iii]

The Daily Illinois State Register, a rival of the Journal, probably knew what had gone on. Passengers remembered the couple, the drunken man and the demonstrative nature of their affection, “caresses and kisses, and occasionally varied the programme by laying her head in his lap.” The passengers were amused until they got into the same berth, “until next morning, and their conduct was such as to attract the attention of those who occupied the section adjacent, who next morning remarked upon it to the conductor.”[iv] The conductor, brakeman and porter corroborated this, and the latter had managed to get a peek at her face, when she lifts her veil to eat breakfast he had brought her. He said she was “about twenty eight years of age, finely formed, black hair and black eyes, and very handsome,” wearing three diamond rings. The Journal soon found itself vehemently denying Louisville reports that Robertson had identified the woman was a married Springfield lady. The fact was, said the Journal, that he had not identified the woman, and, moreover, had declared in a chivalric spirit that his honor forbid his ever doing so.[v] This was, notably, not a denial of the identity itself.

The story exploded in the first week of August, nation-wide, and on August 12, the Journal’s 4th page carried a long piece headlined simply “To My Friends.” It was signed “E. L. Baker.”[vi] It is excruciating to read. “Upon my return to Springfield a few days ago, I was met with the painful rumor that it was my wife who was the companion of the Rev. G. H. Robertson in his recent disgraceful conduct upon the cars between Chicago and Louisville.” He noted that some brave soul had shown him a Louisville Ledger article saying that her name had been signed to several telegrams sent to Robertson after the incident. The “revolting idea” of Julia being involved “shocked and grieved me beyond expression,” but he “instinctively knew at once that there could be no just foundation for it. Whatever the circumstances might indicate, my thorough acquaintance with the purity and dignity of her character,” in the sixteen years they had been married, “utterly forbade that I should for a moment give credence to the scandal; and I determined without hesitation to treat it with the silent contempt which I felt so false a charge deserved.”

However, some “kind and sympathizing friends, for whose judgment I have great respect, have suggested perhaps the community would expect some explanations from me.” He explained that the Chicago telegraph office had no record of communication between them at any time, so the article was false. This would itself “be sufficient to relieve her of all suspicion,” said Baker, before finishing the sentence with “but I would also state,” and described her activities at the relevant time. While he was out of town, Julia had been in Chicago—but only for a visit, accompanied by some friends. The case was not looking good, and he knew it — his references to his inconsideration of circumstances, his multiple pointed legal defense, and a travel pattern in his absence that would account for the time this woman was with Robertson. He bravely attempted to set up a preemptive defense:

“Having nothing to conceal, I would further state that up to the time of his late disgraceful count, Mr. Robertson was very highly esteemed by both myself and wife, as a consistent minister and a  Christian gentleman, and since he left Springfield for Louisville, about a year ago, we have both been in the habit occasionally sending him papers and magazines. In this same way, we have also remembered the Rev. G.W F. Birch, her former pastor, since he left Springfield [for Lexington] . . . I would add that, a few weeks ago, we noticed by the papers that Mr. Robertson was to have a summer vacation; that, after my wife’s return from the lake trip, having him her possession several English periodicals which she desired to mail to him, it not knowing his whereabouts and being at that time utterly ignorant of the shocking story in the newspapers covering him, she innocently sent a telegram to the postmaster at Louisville, asking for Mr. Robert’s present post office address. That telegram, of course, reached Louisville in the midst of the excitement.”

It had all been an unfortunate coincidence. Baker inferred that the postmaster believed it to be part of the scandal, and “instead of replying to it, turned it over to the public.” “That, I am convinced, is the foundation, and the only foundation, for the cure association of the name of my wife with the scandalous affair.” It was really not looking good, and it looked worse by the need to include the following: “For more than two years, as her friends in this city know, my wife has been suffering from a painful and distressing uterine disease which has unfitted her not only for society but for any physical exertion, and most of the time considered her as an invalid at home. For the last year or more she has been under the medical treatment of Dr. Botford, of Chicago, and that was the object of her visit to the city at the time the dreadful story was thus falsely published against her. May God pity her in the anguish which overwhelms her from so unjust an accusation. . . . More than ever , as I now see her crushed by the blow, is she dear and precious to me, and she shall not need for all the poor comfort and tender sympathy which I can give her.”

His defense of his of his wife was followed by Mrs. Robertson’s defense of her husband. He was “the victim of a designing woman,” she wrote to a “prominent gentleman” in Troy, New York, evidently with the intention he leak the letter. She admitted he sometimes drank too much, but had no history of “licentiousness” before he got involved with a “most dangerously fascinating” Springfield woman. This woman was “by no means lewd,” but one of “respectable connections, wealth and position,” and was now writing and appealing to Mrs. Robertson’s womanly feeling, begging that she not be exposed. Mrs. Robertson did not name her, but said her behavior was not new. While they lived in Springfield, the woman had asked him to call on her because she “was in trouble.” He was “suspicious of her,” particularly given that she was not a member of his church—he told her to seek counsel there. Her requests became “so frequent and urgent” that Mrs. Robertson told him to go, and he “afterward fell before the intrigues of this captivating woman.” However, Mrs. Robertson did not feel that he was “criminally associated” with her, and therefore it was her duty to rise to his defense.[vii]

Dr. Robertson, under intense scrutiny, pointedly said that he could “prove beyond cavil” the woman’s identity, but he would “never be guilty of such dishonor, unless drive to the point where honor demands its revelation.” The Cincinnati Enquirer was not impressed. “What point of honor,” is asked, could compel him to do so? His “high sounding show of manly honor is on par with his other doings. It is done for a purpose. Hidden beneath the chivalric words is a threat . . .” He was warning her and her friends against a plan he had learned of—to hire a Chicago prostitute to come forward and take the blame, which his circle found “perfectly horrifying.” “The idea that it should go out to the world” that he had been in the company of “an ordinary scarlet woman was too much.” He had the chivalric concept of honor to stand on, but this source of pride would disappear if the woman had forfeited her right to it. “It was bad enough to fall over a first-class precipice, but to go to the bad like every-day mortals was not to be thought of. Hence, the paragraph which, in plain English, is ‘dare to sully the name of Robertson by associating with it the name of a common courtesan and your identity, which I can prove beyond all cavil, will be proclaimed to the world.’” [viii]

In September, Robertson denied several of the charges made against him in a church inquiry. “I confess that I went to Chicago  . . .to visit a woman not my wife— a woman of the very highest respectability and social position. I went at her very urgent solicitation, after twice refusing, and with the assurance that my expenses should be paid. I went to give the spiritual advice in consolation which she asked and which she assured me she very greatly needed. I had no other purpose in going . . . I remember and acknowledge meeting this woman . . . and starting with her on railway sleeping coach for Louisville; but I do not remember how we reached the depot, neither what persuasion she used to gain my consent to her coming.” He had no recollection of the “flagrant act” of sharing a sleeping coach with her, and denied it. Implying he blacked out, he said he was not consciously guilty of lascivious behavior, as he had never for a moment intended it. He expressed his deep disgrace before his church and wife, and the horror he had caused.[ix] The controversy raged until the end of the year, and the St. Louis Globe, in an editorial titled “The Unprotected Male,” lamented that “Poor Robertson is forever disgraced, while his destroyer, who is beautiful, accomplished and wealthy, and moves in the very highest circles goes unscathed.”[x]

Still, most papers did not let him off the hook, with one warning that he should not be transferred to another congregation “with a set of whitewashing credentials.” While the “lenient” congregation was rallying around him, one paper pointed to the case of a popular Cincinnati minister caught in adultery, who promised to reform “and was placed in charge of a young ladies’ seminary, where the result expected by everybody but the church authors was realized, to the grief and disgrace of several families.”[xi]

Sexual misconduct among clergy was an unfortunately familiar situation—particularly in Springfield. A Massachusetts paper had received a letter that said the people of Springfield “appear to know all about the matter,” which had found its way east. “The lady in question is beautiful, accomplished and fascinating. She is connected with the first families of the State, and has often graced the White House,” during “a former administration. But she ‘hath a devil,’ and the peculiarity of this devil is that it goes out after ministers—pastors of the church to which she belongs. She has been in the habit of sending for them, in the absence of her husband, to ‘administer spiritual consolation;’ and Mr. Robertson is the third minister who has left the church in Springfield which she attends on her account.”[xii]

Conclusion

I extensively quote the press to show just how severe and widespread the embarrassment was to the Edwards family. For people of their status, it was almost the worst possible ordeal. I think they were unfazed by most things by the time Mary Lincoln was committed a few years later, and this goes a long way to explain how they so charitably and placidly handled her from 1875-1882. Mary and Robert must have been aware of the scandal, but it does not seem they were in contact with the Edwardses.

While we don’t know the whole story of what options were explored, her friends and relatives did not have her institutionalized to rid themselves of the embarrassment and clear themselves of responsibility, despite the clear danger she posed to everyone’s reputation. Instead, they left for Argentina, where Baker’s brother already worked. It was probably fairly easy to secure, and the Springfield papers pretended that it had never happened. St. Louis newspapers, which had always been antagonistic, also dropped the story. Perhaps much of the drama related to rivalries among editors, and when Baker exited the scene, they backed off. The Bakers’ departure coincided with a complete re-organization of the Journal, which still made friendly references to Baker, and occasionally published his letters about Argentina. The Edwardses kept in contact with their daughter, and her sons sometimes went to stay with their parents (they had a young daughter with them). They put money in trust for her, and never pretended that she did not exist. It is unlikely that they were socially ostracized, but they were humbled.

More specifics on Julia’s condition are likely revealed by a note written by Elizabeth Edwards to Robert Lincoln, presumably some time shortly after Mary was institutionalized in 1875.

For background information, it is help to know that between Lincoln’s 1860 nomination and 1861 inauguration, Elizabeth was not in Springfield. She was staying with a sick son in Boston, but seems to have stayed away for other reasons. “The delicacy of my health though not apparent to my family, so affected my nervous system, that I was fully sensible that my presence at home could furnish neither pleasure or aid,” she wrote her son-in-law. What this means is unclear, but Julia was pregnant with her second child, not long after welcoming her first child. She was homesick for Lewis, but told him “I know that Julia is surrounded by kind friends who for her sake, and mine will hover around her in trouble – but if she should feel that her mother is necessary to her, a summons would be speedily answered.”[xiii] They named their new baby Elizabeth Edwards Baker, but she died at age six months, shortly before all of them went to Washington with the Lincolns.

Elizabeth’s note to Robert refers only to “my daughter,” no name given, but it seems to be about Julia. There was no explicit reference to any scandals, but Elizabeth explained that “insanity, although a new feature, in our family history, first appeared within my knowledge, in the case of my own daughter, at the early age of thirteen. For six months, “she was so decidedly flighty, as to be closely guarded,” and “At no time has she ever been natural in her demeanor.” Following the birth of each of her children, “the same symptoms were shown, and severely felt, particularly by her husband, and myself.[xiv]”

Footnotes

[1] She stopped by New York City with her husband, behaved extravagantly and apparently otherwise questionably, as Henry Wikoff asked her about it. J. K. C. Forrest remembered that the beautiful “Mrs. Baker” suddenly disappeared from the White House.

[2] Mrs. William H. Bailhache (née Ada Brayman) to Truman Bartlett, Coronado, Colorado, 4 July 1912, Truman Bartlett Papers, Boston University. Thank you to Michael Burlingame for calling this to my attention.

*I’m aware the footnote numbering is off—substack doesn’t handle footnotes well.

[i] Stephen William Berry, House of Abraham: Lincoln and the Todds, a Family Divided by War, 1st Mariner Books ed (Boston: Mariner Books, 2009), 187.

[ii] Cincinnati Commercial Tribune, July 30, 1872.

[iii] Ibid; Daily Missouri Democrat, August 7, 1872.

[iv] “Dr. Robertson/Full Account Of His Fall/What The Train Men Saw/A Most Shameful Exhibition/Statements of Accused/The Mysterious Female/Etc.,  Etc., Etc.,” Louisville Ledger, July 24, 1872, reprinted in Daily Illinois State Journal, July 26, 1872.; “Another Clerical Scandal,” Chicago Times, reprinted in Daily Illinois State Journal, July 25, 1872.

[v] Daily Illinois State Journal, August 8, 1872.

[vi] E. L. Baker, “To My Friends,” Daily Illinois State Journal, August 12, 1872.

[vii ] Troy Times, August 31, 1872, quoted in Cincinnati Daily Enquirer, September 5, 1872.

[viii] Cincinnati Daily Enquirer, August 22, 1872.

[ix] Albany Argus, September 24, 1872.

[x] Quincy Daily Whig, November 5, 1872.

[xi] Daily Missouri Democrat, August 7, 1872.

[xii] Correspondence of the Springfield Republican, “That Woman/Dr. Robertson’s Tempter—A Mania for Ministers,” reprinted in Chicago Post, December 10, 1872.

[xiii] Elizabeth Edwards to Edward L. Baker, Jr., March 4, 1860. The curious reference Elizabeth makes to Julia, that her “back from irritants is scarred its length,” may mean that she was physically restrained during the first episode. But this does not seem to have been repeated. That they took her to Washington suggests that her demeanor could not have been that off-putting, and also that they were willing to risk a lot to allow her full freedom. Her comments seem to suggest post-partum psychosis or mania, and also that she believes there is a biological basis, that there are many degrees of insanity, and that this is a “new” issue. Elizabeth Edwards to Edward L. Baker, Jr., March 4, 1860.

[xiv] Note from Elizabeth Edwards to Robert Lincoln. http://contentdm.acpl.lib.in.us/digital/collection/p16089coll38/id/7022/rec/165. The only indications that people suspected mental illness are the strange passage in Rose Greenhow’s book about her niece’s odd behavior and the remarks of Joseph Howard, Jr. A journalist who accompanied the party to Washington, Howard repeatedly made vague comments about the difficulties caused by Mary’s female relatives, and he remembered a “Mrs. Baker” among the party.

1. https://www.illinoistimes.com/arts-culture/lincolns-scandalous-nephew-11443313